Part 2: tune MobileNet_v2 feature extractor to my space

print(Path().cwd())
os.chdir(L_ROOT)
print(Path().cwd())

/home/ergonyc/Projects/Project2.0/snkr-finder/nbs
/home/ergonyc/Projects/Project2.0/snkr-finder

filename = ZAPPOS_DF_SIMPLIFIED # "zappos-50k-simplified"
df = pd.read_pickle(f"data/{filename}.pkl")

mobilenet v2

We will use Google's mobileNetV2 trained on ImageNet loaded from torchvision to embed our sneakers into a feature space.

decapitate mobilnet_v2 (neuter)

Because we simply want to collect the features output from the model rather than do classification (or some other decision) I replaced the clasiffier head with a simple identity mapper. The simple Identity nn.Module class makes this simple.

Finally, since we are calculating the features, or embedding over 30k images with the net lets load the computations onto our GPU. We need to remember to do this in evaluation mode so Batch Norm / Dropout layers are disabled. [I forgot to do this initally and lost hours trying to figure out why i wasn't getting consistent results]. Setting param.requires_grad = False saves us memory since we aren't going to fit any weights for now, and protects us in case we forget to do a with torch.no_grad() before inference.

Later when we use the full FastAI API this should all be handled elegantly behind the scenes

This function neuters the mobilenet v2 and pools the output with avg/max across the spatial dimension using torch calls.

This function neuters the fastai resnet18 and pools the output with avg/max across the spatial dimension using low-level fastai api torch wrappers.

Fully using the fastAI API makes it even simpler!

I just need to figure out how / if to pool the spatial dimensions of the feature extractor... and double check what the mvnetv2 actuall does in my instattiation.

We should probably set up our dataloaders to load all of the data into the 'valid'. Then we don't need to worry about the Resize() function doing random stuff.

This creates the right indices to get an empty dls.train and all images in dls.valid, but its not happy creating an empty train dls...

IndexSplitter(df.index.values.tolist())(df),df.shape

Turns out this feature of Resize results in a random crop for training contexts, and is a bit of a bug for what I'm doing here. The rest of the special features of the fastai Resize has compared to coding it with torch makes it worth these hacks. For now I'll subclass a FeatsResize and replace the before_call() method which performs the split_idx voodoo.

sz=IMG_SIZES['small']
device = get_cuda()
batch_size = 64

dls = get_zap_feats_dataloaders(df,batch_size,sz,device)
#model = get_mnetV2_feature_net(to_cuda=True)

df_f = get_feats_df(dls,model)
df_f.head()

packing 432 batches into dataframe

IMG_SIZES

{'small': 128, 'medium': 160, 'large': 224}

#     f = open(filepath, "wb")
#     pickle.dump(item_to_save, f)
#     f.close()


# def load_pickle(filepath):
#     infile = open(filepath, "rb")
#     item = pickle.load(infile)
#     infile.close()
#     return item

model = create_cnn_featurenet(torchvision.models.mobilenet_v2,to_cuda=True)

save_featsXsize(df,model)
mnet_df =  collate_featsXsize(df,model.name)

128
packing 432 batches into dataframe
160
packing 432 batches into dataframe
224
packing 432 batches into dataframe
128
160
224

model = create_cnn_featurenet(resnet18,to_cuda=True)
save_featsXsize(df,model)

rnet_df =  collate_featsXsize(df,model.name)

128
packing 432 batches into dataframe
160
packing 432 batches into dataframe
224
packing 432 batches into dataframe
128
160
224

If we've already calculated everything just load it.

SANITY CHECK:

Just want to chack that we can we extract single features that match those we just calculated.

# query_image = "Shoes/Sneakers and Athletic Shoes/Nike/7716996.288224.jpg"
df.loc[df.path==QUERY_IM,['path','classes_md']]

The DataBlock performed a number of processing steps to prepare the images for embedding into the MobileNet_v2 space (1280 vector). Lets confirm that we get the same image and MobileNet_v2 features.

First, lets wrap our model in a function which will make sure we are in inference mode and things are sent to the cpu.

Now, we need to transform our inference image into a tensore properly sized and normalized for the model. Something link this:

That works okay, and just wrapping it in a simple function makes it pretty simple...

test_feats = get_mnet_feature(mnetv2,query_t)
test_feats.shape

torch.Size([1, 2560])

Buuuuuut, a FastAI Pipeline is just a couple lines of code. Now we're cookin'. Note that Normalize.from_stats usually is executed on batches loaded into the GPU so we need to make sure it has the cuda=False flag set. I suppose we could be doing massively parallel inference and want this pipeline on a GPU at some point, but typically we'll want cpus powered transform.

mnet1 = get_mnetV2_feature_net()
query_t1 = load_and_prep_sneaker(images_path/QUERY_IM)
test_feats1 = get_mnet_feature(mnet1,query_t1)


mnet2 = create_cnn_featurenet(torchvision.models.mobilenet_v2)
query_t2 = load_and_prep_tf_pipe(images_path/QUERY_IM)
test_feats2 = get_mnet_feature(mnet2,query_t2)


#test_feats1.mean(),test_feats2.mean(),(test_feats1-test_feats2).max(),
#PILImage.create((query_t1-query_t2).squeeze())

qt = load_and_prep_tf_pipe(images_path/QUERY_IM)
test_feats2 = get_mnet_feature(mnetv2,qt)
test_feats2.shape,

(torch.Size([1, 2560]),)

Now I have the "embeddings" of the database in the mobileNet_v2 output space. I can do a logistic regression on these vectors (should be identical to mapping these 1000 vectors to 4 categories (Part 3)) but I can also use an approximate KNN in this space to run the SneakerFinder tool.

k-Nearest Neighbors: a proxy for "similar"

I'll start with a simple "gut" test, and point out that thre realy isn't a ground truth to refer to. Remember that the goal of all this is to find some shoes that someone will like, and we are using "similar" as the aproximation of human preference.

Lets use our previously calculated sneaker-features and inspect that the k- nearest neighbors in our embedding space are feel or look "similar".

Personally, I like Jordans so I chose this as my query_image: Sample Jordan

Lets take a quick look at the neighbors according to our list:

num_neighs = 9

knns, reducers = get_neighs_and_reducers(df,num_neighs=num_neighs)



neighs = knns[0]
distance, nn_index = neighs.kneighbors(test_feats, return_distance=True)    


dist = distance.tolist()[0]

sm
128
features_sm
md
160
features_md
lg
224
features_lg

filename = f"data/{model.name}-knnXsize_nn{num_neighs}.pkl"
dump_pickle(filename,knns)

filename = f"data/{model.name}-umapXsize.pkl"
dump_pickle(filename,reducers)

num_neighs = 9
model = create_cnn_featurenet(resnet18,to_cuda=True)

filename = f"data/knnXsize_nn{num_neighs}.pkl"
knns = load_pickle(filename)

filename = f"data/umapXsize.pkl"
reducers = load_pickle(filename)

paths_df = df[['path','classes_sm','classes_md','classes_lg']]
neighbors = paths_df.iloc[nn_index.tolist()[0]].copy()


query_t2 = load_and_prep_tf_pipe(images_path/QUERY_IM)
test_feats2 = get_mnet_feature(model,query_t2)


neighs = knns[0]
distance, nn_index = neighs.kneighbors(test_feats2, return_distance=True)    
dist = distance.tolist()[0]

images = [ PILImage.create(images_path/f) for f in neighbors.path] 
#PILImage.create(btn_upload.data[-1])
for im in images:
    display(im.to_thumb(IMG_SIZE,IMG_SIZE))

similar_images =  get_similar_images( paths_df,model,knns)

sm
128
features_sm

md
160
features_md

lg
224
features_lg

plot_sneak_neighs(similar_images)

similar_images2 = []
for i,sz in enumerate(IMG_SIZES):
    print(SIZE_ABBR[sz])
    print(IMG_SIZES[sz])
    
    features = f"features_{SIZE_ABBR[sz]}"
    print(features)

    query_t = load_and_prep_sneaker(QUERY_IM2,IMG_SIZES[sz])
    query_f = get_mnet_feature(mnetv2,query_t)
    
    similar_images2.append( query_neighs(query_f, knns[i], paths, images_path, show=False) )

    im = PILImage.create(QUERY_IM2)
    display(im.to_thumb(IMG_SIZES[sz]))
    
plot_sneak_neighs(similar_images2)

sm
128
features_sm

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-75-b96d9ed64c20> in <module>
     10     query_f = get_mnet_feature(mnetv2,query_t)
     11 
---> 12     similar_images2.append( query_neighs(query_f, knns[i], paths, images_path, show=False) )
     13 
     14     im = PILImage.create(QUERY_IM2)

NameError: name 'paths' is not defined

visualize the embedding: PCA + UMAP

# first simple PCA
pca = PCA(n_components=2)

for i,sz in enumerate(IMG_SIZES):
    print(SIZE_ABBR[sz])
    print(IMG_SIZES[sz])
    
    features = f"features_{SIZE_ABBR[sz]}"
    print(features)
    
    data = df[['Category',features]].copy()

    db_feats = np.vstack(data[features].values)

    # PCA
    pca_result = pca.fit_transform(db_feats)
    data['pca-one'] = pca_result[:,0]
    data['pca-two'] = pca_result[:,1] 
    print(f"Explained variation per principal component (sz{sz}): {pca.explained_variance_ratio_}")

    smpl_fac=.5
    #data=df.reindex(rndperm)

    plt.figure(figsize=(16,10))
    sns.scatterplot(
        x="pca-one",
        y="pca-two",
        hue="Category",
        palette=sns.color_palette("hls", 4),
        data=data.sample(frac=smpl_fac),
        legend="full",
        alpha=0.3
    )
    plt.savefig(f'PCA 2-D sz{sz}')
    plt.show()
    
    
    # get the UMAP on deck
    embedding = reducers[i].transform(db_feats)
    
    data['umap-one'] = embedding[:,0]
    data['umap-two'] = embedding[:,1] 

    plt.figure(figsize=(16,10))
    sns.scatterplot(
        x="umap-one",
        y="umap-two",
        hue="Category",
        palette=sns.color_palette("hls", 4),
        data=data.sample(frac=smpl_fac),
        legend="full",
        alpha=0.3
    )
    plt.gca().set_aspect('equal', 'datalim')
    plt.title(f'UMAP projection of mobileNetV2 embedded UT-Zappos data (sz{sz})', fontsize=24)
    plt.savefig('UMAP 2-D sz{sz}') 
    plt.show()

sm
128
features_sm
Explained variation per principal component (szsmall): [0.11515645 0.06882259]

md
160
features_md
Explained variation per principal component (szmedium): [0.12368079 0.07145122]

lg
224
features_lg
Explained variation per principal component (szlarge): [0.12368079 0.07145125]

fn = df.path.values
type(db_feats)

snk2vec = dict(zip(fn,db_feats))

snk2vec[list(snk2vec.keys())[0]]

embedding = get_umap_embedding(db_feats)
snk2umap = dict(zip(fn,embedding))

GLM: Logistic regression on the mobilnet_v2 features: validation and categorization

Note that because we're not really doing parameter fitting, we'll ignore the validation set and add it to training.

filename = f"zappos-50k-{model.name}-features_sort_3"

df = pd.read_pickle(f"data/{filename}.pkl")

#Display Confusion Matrix
X_test = np.vstack(df[df.t_t_v=='test']['features_lg'])
y_test = np.vstack(df[df.t_t_v=='test']['Category']).flatten()

# use validate and train for training (no validation here)
X_train = np.vstack(df[df.train | df.validate]['features_lg'])
y_train = np.vstack(df[df.train | df.validate]['Category']).flatten()


clf_log = LogisticRegression(C = 1, multi_class='ovr', max_iter=2000, solver='lbfgs')
clf_log.fit(X_train, y_train)
log_score = clf_log.score(X_test, y_test)
log_ypred = clf_log.predict(X_test)

log_confusion_matrix = confusion_matrix(y_test, log_ypred)
print(log_confusion_matrix)

disp = heatmap(log_confusion_matrix, annot=True, linewidths=0.5, cmap='Blues')
plt.savefig('log_Matrix.png')


plt.figure(figsize=(16,16))


# Plot non-normalized confusion matrix
titles_options = [("Confusion matrix, without normalization", None),
                  ("Normalized confusion matrix", 'true')]

/home/ergonyc/anaconda3/envs/fastai/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/ergonyc/anaconda3/envs/fastai/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(
/home/ergonyc/anaconda3/envs/fastai/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(

[[1227   16    2   16]
 [  15 1229   26   59]
 [   5   44   85    4]
 [  36   85    4 1290]]

/home/ergonyc/anaconda3/envs/fastai/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:762: ConvergenceWarning: lbfgs failed to converge (status=1):
STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.

Increase the number of iterations (max_iter) or scale the data as shown in:
    https://scikit-learn.org/stable/modules/preprocessing.html
Please also refer to the documentation for alternative solver options:
    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression
  n_iter_i = _check_optimize_result(

<Figure size 1152x1152 with 0 Axes>

class_names = df.Category.unique()

from sklearn.metrics import plot_confusion_matrix

for title, normalize in titles_options:
    disp = plot_confusion_matrix(clf_log, X_test, y_test,
                                 display_labels=class_names,
                                 cmap=plt.cm.Blues,
                                 normalize=normalize)
    disp.ax_.set_title(title)

    print(title)
    print(disp.confusion_matrix)

plt.savefig('log_Matrix2.png')

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-27-06a150b5b490> in <module>
----> 1 class_names = df.Category.unique()
      2 
      3 from sklearn.metrics import plot_confusion_matrix
      4 
      5 for title, normalize in titles_options:

NameError: name 'df' is not defined

transfer learning

by hand

Here's how we could would do transfer learning "by hand":

1. load the pretrained network
2. create a new linear classifier e.g. `nn.Linear(num_ftrs, n_categories)`
3. _freeze_ the parameters by setting `param.requires_grad=False` (NOTE, that to actually work we need to NOT freeze batchnorm layers)
4. create a training loop (or send to a fastai learner)

the fastai way

With the fastai API we can simply use cnn_learner with the name of the architecture, and everything else is semi-automatic. e.g.

1. load the arcitecture and trained weights
3. creating a classifier "head"
3. setting up the parameters for freezing (avoiding batchnorms)

Note that the mobilenet V2 architecture is NOT part of the API so we'll need to get the weights and arch from torchvision, and hack in the the splitter and cut points.

I'm also going to wrap the dataframe -> DataBlock -> dataloaders in some convenience functions to make the whole shebang just a few lines.

filename = ZAPPOS_DF_SIMPLIFIED # "zappos-50k-simplified"
df = pd.read_pickle(f"data/{filename}.pkl")
df = prep_df_for_datablocks(df)

df.shape[0]/32

862.9375

#dls = get_zappos_cat_dataloaders(df)
dls2 = get_zappos_cat_dataloaders()
rnet_learn = cnn_learner(dls2, resnet18, metrics=error_rate)
dls.show_batch()

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-90-4a5f94a19cb4> in <module>
      3 dls2 = get_zappos_cat_dataloaders()
      4 rnet_learn = cnn_learner(dls2, resnet18, metrics=error_rate)
----> 5 dls.show_batch()

NameError: name 'dls' is not defined

lr_min,lr_steep = rnet_learn.lr_find()
mlr = .5*(lr_min+lr_steep)
#geometric mean
gmlr = torch.tensor([lr_min,lr_steep]).log().mean().exp().tolist()
lr_min,lr_steep,mlr,gmlr

(0.006918309628963471,
 0.0004786300996784121,
 0.0036984698643209414,
 0.0018197011668235064)

rnet_learn.fine_tune(2, base_lr=gmlr,freeze_epochs=1)
rnet_learn.show_results()

filename = 'rnet18_transfer-feb20_1x2b'
rnet_learn.save(filename)

Path('models/rnet18_transfer-feb20_1x2b.pth')

rnet_learn.export(fname=filename)

freeze_epochs,epochs = 4,2
lr_min,lr_steep = rnet_learn.lr_find()
#geometric mean
gmlr = torch.tensor([lr_min,lr_steep]).log().mean().exp().tolist()

rnet_learn.fine_tune(epochs, base_lr=gmlr,freeze_epochs=freeze_epochs)
rnet_learn.show_results()

filename = f'rnet18_transfer-feb20_{freeze_epochs}x{epochs}b'
rnet_learn.save(filename)

rnet_learn.save('rnet18_transfer-fep20_1x2')
rnet_learn.export(fname=filename)

Path('models/rnet18_transfer-fep20_1x2.pth')

mnet_learn = cnn_learner(dls,torchvision.models.mobilenet_v2, n_out=4,
                    pretrained=True,metrics=error_rate)

lr_min,lr_steep = mnet_learn.lr_find()
mlr = .5*(lr_min+lr_steep)
#geometric mean
gmlr = torch.tensor([lr_min,lr_steep]).log().mean().exp().tolist()
lr_min,lr_steep,mlr,gmlr

(0.006918309628963471,
 0.0008317637839354575,
 0.003875036706449464,
 0.0023988327011466026)

freeze_epochs,epochs = 4,2
mnet_learn.fine_tune(epochs, base_lr=gmlr,freeze_epochs=freeze_epochs)
mnet_learn.show_results()

filename = f'mnet18_transfer-feb20_{freeze_epochs}x{epochs}b'
rnet_learn.save(filename)
rnet_learn.export(fname=filename)

Path('models/mnet_transfer-fep20_1x2.pth')

freeze_epochs,epochs = 2,1

mnet_learn = cnn_learner(dls,torchvision.models.mobilenet_v2, n_out=4,
                    pretrained=True,metrics=error_rate)

lr_min,lr_steep = mnet_learn.lr_find()
gmlr = torch.tensor([lr_min,lr_steep]).log().mean().exp().tolist() #geometric mean

mnet_learn.fine_tune(epochs, base_lr=gmlr,freeze_epochs=freeze_epochs)
mnet_learn.show_results()

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-49-dd7beba6fc75> in <module>
      4                     pretrained=True,metrics=error_rate)
      5 
----> 6 lr_min,lr_steep = mnet_learn.lr_find()
      7 gmlr = torch.tensor([lr_min,lr_steep]).log().mean().exp().tolist() #geometric mean
      8 

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/callback/schedule.py in lr_find(self, start_lr, end_lr, num_it, stop_div, show_plot, suggestions)
    220     n_epoch = num_it//len(self.dls.train) + 1
    221     cb=LRFinder(start_lr=start_lr, end_lr=end_lr, num_it=num_it, stop_div=stop_div)
--> 222     with self.no_logging(): self.fit(n_epoch, cbs=cb)
    223     if show_plot: self.recorder.plot_lr_find()
    224     if suggestions:

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in fit(self, n_epoch, lr, wd, cbs, reset_opt)
    209             self.opt.set_hypers(lr=self.lr if lr is None else lr)
    210             self.n_epoch = n_epoch
--> 211             self._with_events(self._do_fit, 'fit', CancelFitException, self._end_cleanup)
    212 
    213     def _end_cleanup(self): self.dl,self.xb,self.yb,self.pred,self.loss = None,(None,),(None,),None,None

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_fit(self)
    200         for epoch in range(self.n_epoch):
    201             self.epoch=epoch
--> 202             self._with_events(self._do_epoch, 'epoch', CancelEpochException)
    203 
    204     def fit(self, n_epoch, lr=None, wd=None, cbs=None, reset_opt=False):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_epoch(self)
    194 
    195     def _do_epoch(self):
--> 196         self._do_epoch_train()
    197         self._do_epoch_validate()
    198 

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_epoch_train(self)
    186     def _do_epoch_train(self):
    187         self.dl = self.dls.train
--> 188         self._with_events(self.all_batches, 'train', CancelTrainException)
    189 
    190     def _do_epoch_validate(self, ds_idx=1, dl=None):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in all_batches(self)
    164     def all_batches(self):
    165         self.n_iter = len(self.dl)
--> 166         for o in enumerate(self.dl): self.one_batch(*o)
    167 
    168     def _do_one_batch(self):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in one_batch(self, i, b)
    182         self.iter = i
    183         self._split(b)
--> 184         self._with_events(self._do_one_batch, 'batch', CancelBatchException)
    185 
    186     def _do_epoch_train(self):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _with_events(self, f, event_type, ex, final)
    158 
    159     def _with_events(self, f, event_type, ex, final=noop):
--> 160         try: self(f'before_{event_type}');  f()
    161         except ex: self(f'after_cancel_{event_type}')
    162         self(f'after_{event_type}');  final()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/learner.py in _do_one_batch(self)
    175         if not self.training or not len(self.yb): return
    176         self('before_backward')
--> 177         self.loss_grad.backward()
    178         self._with_events(self.opt.step, 'step', CancelStepException)
    179         self.opt.zero_grad()

~/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    212         from torch.overrides import has_torch_function, handle_torch_function
    213         if type(self) is not Tensor and has_torch_function(relevant_args):
--> 214             return handle_torch_function(
    215                 Tensor.backward,
    216                 relevant_args,

~/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/overrides.py in handle_torch_function(public_api, relevant_args, *args, **kwargs)
   1058         # Use `public_api` instead of `implementation` so __torch_function__
   1059         # implementations can do equality/identity comparisons.
-> 1060         result = overloaded_arg.__torch_function__(public_api, types, args, kwargs)
   1061 
   1062         if result is not NotImplemented:

~/anaconda3/envs/fastai/lib/python3.8/site-packages/fastai/torch_core.py in __torch_function__(self, func, types, args, kwargs)
    323         convert=False
    324         if _torch_handled(args, self._opt, func): convert,types = type(self),(torch.Tensor,)
--> 325         res = super().__torch_function__(func, types, args=args, kwargs=kwargs)
    326         if convert: res = convert(res)
    327         if isinstance(res, TensorBase): res.set_meta(self, as_copy=True)

~/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/tensor.py in __torch_function__(cls, func, types, args, kwargs)
    993 
    994         with _C.DisableTorchFunction():
--> 995             ret = func(*args, **kwargs)
    996             return _convert(ret, cls)
    997 

~/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    219                 retain_graph=retain_graph,
    220                 create_graph=create_graph)
--> 221         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    222 
    223     def register_hook(self, hook):

~/anaconda3/envs/fastai/lib/python3.8/site-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
    128         retain_graph = create_graph
    129 
--> 130     Variable._execution_engine.run_backward(
    131         tensors, grad_tensors_, retain_graph, create_graph,
    132         allow_unreachable=True)  # allow_unreachable flag

RuntimeError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 7.79 GiB total capacity; 3.68 GiB already allocated; 20.50 MiB free; 3.71 GiB reserved in total by PyTorch)

filename = f'mnet18_transfer-feb20_{freeze_epochs}x{epochs}b'
rnet_learn.save(filename)
rnet_learn.export(fname=filename)

Path('models/mnet_transfer-fep20_2x4.pth')

interp = Interpretation.from_learner(mnet_learn)
interp.plot_top_losses(9, figsize=(15,10))

rnet_learn=load_learner('rnet_transfer-feb20_1x2a')

interp = Interpretation.from_learner(rnet_learn)
interp.plot_top_losses(9, figsize=(15,10))

interp.plot_top_losses()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-93-99f2bc1e1042> in <module>
----> 1 interp.plot_top_losses()

TypeError: plot_top_losses() missing 1 required positional argument: 'k'

export

from nbdev.export import notebook2script
notebook2script()

Converted 00_core.ipynb.
Converted 01a_zappos_data.ipynb.
Converted 01b_scraped_data.ipynb.
Converted 02_model.ipynb.
Converted 03a_cvae.ipynb.
Converted index.ipynb.

	path	classes	features
0	Boots/Ankle/A. Testoni/7965307.5291.jpg	0	[6.1511936, 0.49706665, 1.8060436, 1.4014965, 1.1964728, 2.0683293, 6.2304153, 0.0, 1.029815, 1.4007341, 5.218934, 5.7439003, 4.9946513, 3.3029056, 0.0, 2.3674772, 0.0, 1.5000327, 0.7199159, 7.3761997, 0.0, 0.5676668, 10.539913, 4.050178, 0.16800794, 7.028089, 6.665336, 2.5258596, 0.0, 0.41142434, 4.386546, 2.7574868, 2.990525, 0.2352151, 0.24582782, 2.768759, 0.0, 0.6105607, 5.9695935, 2.828486, 1.4011102, 6.8150244, 3.9221385, 0.92679423, 2.9838753, 6.3029475, 5.5856466, 1.2300881, 0.0, 2.1340559, 0.0, 5.5748386, 0.0, 6.428156, 5.395323, 2.0941038, 0.89731944, 0.28486687, 3.5300581, 0.0,...
1	Boots/Ankle/A. Testoni/7999255.363731.jpg	1	[2.548153, 1.0572189, 0.0, 2.22272, 0.0, 0.0, 0.4445022, 0.0, 4.7316513, 4.176343, 3.1725342, 1.4400064, 3.6576366, 1.242915, 5.284669, 2.0685205, 0.0, 0.0, 1.4227625, 3.2478065, 2.5648909, 5.909906, 2.8907194, 2.3030896, 0.0, 2.994581, 5.650822, 10.621788, 1.4928083, 0.55081075, 1.0814705, 1.7987698, 3.860276, 1.45635, 2.9508052, 0.24207915, 0.54251695, 1.2813618, 1.7510852, 0.9131382, 1.6330549, 5.0421476, 6.935538, 4.8825827, 2.328521, 1.0601387, 2.7579157, 0.0, 0.4695269, 2.5846472, 1.3412735, 2.7197359, 0.0, 0.0, 5.078348, 4.448706, 3.0495028, 1.2610846, 4.0939693, 1.297782, 0.0, 3.79...
2	Boots/Ankle/A. Testoni/8000978.364150.jpg	2	[1.1727737, 1.8935859, 0.41519675, 3.5834513, 0.0, 0.79798126, 0.84522206, 2.0391512, 6.3991275, 2.2314568, 3.077018, 4.6675735, 3.1369753, 2.0673757, 0.017585099, 1.6838398, 0.27630857, 0.0, 1.5110508, 1.4410914, 0.59388644, 1.1969396, 1.5390207, 5.0867023, 1.648392, 3.55574, 4.2422595, 3.1608982, 0.98261887, 0.3479449, 1.9060917, 0.0, 3.8188927, 0.030002007, 2.1720817, 1.0492408, 3.5889754, 2.7572808, 3.5072227, 0.4513713, 2.4878974, 9.653975, 5.5371513, 0.2988862, 1.9339348, 3.9931672, 2.1140728, 0.0, 1.3220974, 1.0875705, 0.16992529, 2.5798419, 0.0, 1.5891038, 5.1934485, 7.4684424, 3.8...
3	Boots/Ankle/AIGLE/8113228.1897.jpg	3	[3.871399, 3.3612769, 0.61680937, 4.8329973, 0.0, 1.6733338, 0.24707784, 0.65770924, 4.4510202, 4.881236, 1.99946, 2.9580646, 1.7309498, 3.6685898, 0.9332123, 4.830799, 0.0, 3.6579316, 0.0, 1.5166997, 0.0, 0.6227035, 1.8431097, 3.7361774, 0.0, 5.521512, 4.8245893, 4.005489, 3.490752, 0.0, 1.1609255, 5.109989, 0.9151298, 2.7282424, 0.50677925, 0.6875251, 1.5075024, 2.404952, 4.2298675, 2.1394227, 0.0, 4.6833606, 7.8285127, 0.43900314, 2.7598078, 5.2773113, 2.5236495, 0.0, 1.9644412, 4.6878176, 3.1972182, 1.542216, 0.7499213, 0.0, 5.3819213, 5.046078, 2.8595643, 2.7141469, 6.492911, 0.285495...
4	Boots/Ankle/AIGLE/8113228.1912.jpg	4	[4.6750154, 2.9334877, 0.0091399625, 3.857977, 0.0, 2.4894032, 1.0634216, 0.601744, 4.4283323, 3.5292544, 3.261726, 2.4730198, 3.5346577, 4.713074, 1.6284947, 5.9923744, 0.022872573, 4.793993, 1.3621124, 2.788335, 1.198917, 0.90964204, 0.16350175, 1.5647085, 0.50867623, 3.6006813, 4.650574, 3.7061806, 2.5896811, 0.0, 2.0800378, 3.2416968, 0.8153863, 3.4322462, 0.0, 0.6010052, 1.9733667, 3.35225, 3.718601, 2.0795012, 0.0, 2.71075, 9.500149, 1.8372624, 1.8899909, 2.8282526, 2.7681763, 0.0, 3.811124, 3.9409692, 0.86870706, 3.2808516, 0.0, 0.0, 4.2710075, 5.3361635, 1.6290514, 3.7155895, 7.446...

epoch	train_loss	valid_loss	error_rate	time
0	0.239822	0.206779	0.073386	00:33
1	0.156749	0.161141	0.057815	00:33

epoch	train_loss	valid_loss	error_rate	time
0	0.262603	0.196889	0.070247	00:34
1	0.154782	0.153373	0.056005	00:33

epoch	train_loss	valid_loss	error_rate	time
0	0.274249	0.190135	0.066868	00:42
1	0.166819	0.162267	0.058057	00:42

Part 2: tune MobileNet_v2 feature extractor to my space

mobilenet v2

decapitate mobilnet_v2 (neuter)

get_cuda[source]

get_cpu[source]

class AdaptiveConcatPoolFlat[source]

get_mnetV2_feature_net[source]

get_rnet_feature_net[source]

Fully using the fastAI API makes it even simpler!

create_cnn_featurenet[source]

class FeatsResize[source]

zap_get_x[source]

zap_get_y[source]

zap_get_fname[source]

get_zap_feats_dataloaders[source]

get_all_feats[source]

get_feats_df[source]

save_feats[source]

save_featsXsize[source]

collate_featsXsize[source]

SANITY CHECK:

get_mnet_feature[source]

load_and_prep_sneaker[source]

k-Nearest Neighbors: a proxy for "similar"

get_umap_reducer[source]

get_neighs_and_reducers[source]

query_neighs[source]

get_similar_images[source]

plot_sneak_neighs[source]

visualize the embedding: PCA + UMAP

get_umap_embedding[source]

GLM: Logistic regression on the mobilnet_v2 features: validation and categorization

transfer learning

by hand

transfer_mobilenet_v2[source]

the fastai way

zap_get_x[source]

zap_get_y[source]

get_zappos_datablock[source]

prep_df_for_datablocks[source]

get_zappos_cat_dataloaders[source]

export

`get_cuda`[source]

`get_cpu`[source]

`class` `AdaptiveConcatPoolFlat`[source]

`get_mnetV2_feature_net`[source]

`get_rnet_feature_net`[source]

`create_cnn_featurenet`[source]

`class` `FeatsResize`[source]

`zap_get_x`[source]

`zap_get_y`[source]

`zap_get_fname`[source]

`get_zap_feats_dataloaders`[source]

`get_all_feats`[source]

`get_feats_df`[source]

`save_feats`[source]

`save_featsXsize`[source]

`collate_featsXsize`[source]

`get_mnet_feature`[source]

`load_and_prep_sneaker`[source]

`get_umap_reducer`[source]

`get_neighs_and_reducers`[source]

`query_neighs`[source]

`get_similar_images`[source]

`plot_sneak_neighs`[source]

`get_umap_embedding`[source]

`transfer_mobilenet_v2`[source]

`zap_get_x`[source]

`zap_get_y`[source]

`get_zappos_datablock`[source]

`prep_df_for_datablocks`[source]

`get_zappos_cat_dataloaders`[source]